[PERFSCALE-3943] LACP bonding for perf/scale lab (2) #706

josecastillolema · 2025-09-27T10:48:03Z

No description provided.

josecastillolema · 2025-09-29T14:24:30Z

Ongoing testing in: openshift/release#69765
SNO deployments look fine ✔️

Bastion host

# nmcli con
NAME                 UUID                                  TYPE      DEVICE
Wired connection 1   d223b670-71e3-33b5-9532-c31f77fa811d  ethernet  eno1np0
eno2np1              2dfdcbe6-030d-4061-ac08-92979d58b9e7  ethernet  eno2np1
bond0                44949383-52ad-4e5f-be91-0d75e883cbe4  bond      bond0
bond0-slave-ens1f0   1d6eb689-7536-40d1-a13c-a07f18335a72  ethernet  ens1f0
bond0-slave-ens1f1   393074e4-ed46-4707-b500-477b2a925c19  ethernet  ens1f1
lo                   f3932fd2-b907-40ab-8855-af73dfee5efa  loopback  lo
bond0-slave-eno1np0  db04bb01-eca6-48c5-a5a4-37adb2fa0782  ethernet  --
Wired connection 2   4e92f0b5-fed6-3fd1-b8a0-0e22075fbc3f  ethernet  --
Wired connection 3   ee2fb421-83bd-3b2f-8450-245250de4f9a  ethernet  --
Wired connection 4   f5429206-1761-3586-a64b-577b9898c596  ethernet  --

SNO host

sh-5.1# ovs-vsctl show
b6e31319-37dc-4594-8b44-df31e8576371
    Bridge br-ex
        Port patch-br-ex_f19-h02-000-r640-to-br-int
            Interface patch-br-ex_f19-h02-000-r640-to-br-int
                type: patch
                options: {peer=patch-br-int-to-br-ex_f19-h02-000-r640}
        Port br-ex
            Interface br-ex
                type: internal
        Port bond0
            Interface bond0
                type: system

sh-5.1# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.14.0-427.92.1.el9_4.x86_64

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 40:a6:b7:2a:74:40
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 21
        Partner Key: 12
        Partner Mac Address: 84:c1:c1:47:04:fc

Slave Interface: ens1f0
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:a6:b7:2a:74:40
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 40:a6:b7:2a:74:40
    port key: 21
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: 84:c1:c1:47:04:fc
    oper key: 12
    port priority: 127
    port number: 12
    port state: 63

Slave Interface: ens1f1
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:a6:b7:2a:74:41
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 40:a6:b7:2a:74:40
    port key: 21
    port priority: 255
    port number: 2
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: 84:c1:c1:47:04:fc
    oper key: 12
    port priority: 127
    port number: 6
    port state: 63

josecastillolema · 2025-10-02T10:26:00Z

And bonding working with MNO as well:

# oc get node
NAME               STATUS   ROLES                  AGE   VERSION
f19-h0x-000-r640   Ready    control-plane,master   22h   v1.31.13
f19-h0y-000-r640   Ready    control-plane,master   22h   v1.31.13
f19-h0z-000-r640   Ready    control-plane,master   22h   v1.31.13
f19-h1x-000-r640   Ready    worker                 22h   v1.31.13
f19-h1y-000-r640   Ready    worker                 22h   v1.31.13

# oc debug node/f19-h17-000-r640
Starting pod/f19-h17-000-r640-debug-ngkgf ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.216.8
If you don't see a command prompt, try pressing enter.
sh-5.1# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.14.0-427.92.1.el9_4.x86_64

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: xx
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 21
        Partner Key: 15
        Partner Mac Address: xx

Slave Interface: ens1f0
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:a6:b7:2b:a2:90
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: xx
    port key: 21
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: xx
    oper key: 15
    port priority: 127
    port number: 3
    port state: 63

Slave Interface: ens1f1
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: xx
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: xx
    port key: 21
    port priority: 255
    port number: 2
    port state: 61
details partner lacp pdu:
    system priority: 127
    system mac address: xx
    oper key: 15
    port priority: 127
    port number: 9
    port state: 63

sh-5.1# chroot /host
sh-5.1# ovs-vsctl show
...
    Bridge br-ex
        Port patch-br-ex_f19-h17-000-r640-to-br-int
            Interface patch-br-ex_f19-h17-000-r640-to-br-int
                type: patch
                options: {peer=patch-br-int-to-br-ex_f19-h17-000-r640}
        Port br-ex
            Interface br-ex
                type: internal
        Port bond0
            Interface bond0
                type: system
    ovs_version: "3.5.2-33.el9fdp"

josecastillolema · 2025-10-02T10:27:47Z

/assign @akrzos

josecastillolema · 2025-10-02T10:27:57Z

@akrzos this is ready for review! Thanks

akrzos

I think there is a slight mis-understanding to the indexes of mac addresses provided by the scale lab and the nic names saved in hw_nic_name that we should address.

ansible/roles/bastion-network/tasks/main.yml

akrzos · 2025-10-03T14:22:09Z

ansible/roles/create-inventory/defaults/main/networks.yml

-bond0_port1: 1
-bond0_port2: 2
+# Default behavior uses interfaces at indices 0 and 1 from hw_nic_name mapping
+bond0_port1: 0


I'm a little confused at choosing "index" 0 since that would refer to the public nic in hw_nic_name Is that intentional?

akrzos · 2025-10-03T14:25:08Z

ansible/roles/create-inventory/tasks/main.yml

    - name: MNO - Set controlplane0 bond mac addresses
      set_fact:
-        controlplane0_bond0_macs: "{{ (controlplane0_foreman_data.json.interfaces | selectattr('name', 'eq', bond0_interface1) | first).mac }},{{ (controlplane0_foreman_data.json.interfaces | selectattr('name', 'eq', bond0_interface2) | first).mac }}"
+        controlplane0_bond0_macs: "{{ ocpinventory.json.nodes[1].mac[bond0_port1|int] }},{{ ocpinventory.json.nodes[1].mac[bond0_port2|int] }}"


For mac addresses, the scale lab does not include the public nic in the list of mac addresses so this is likely the reason you may have adjusted bond0_port1 and bond0_port2 back one index. The issue here is the scale lab index and hw_nic_name do not match for mac and nic names. This was intentional at the time hw_nic_name because it was solely for managing what nic name would be choosen for hypervisors.

Signed-off-by: Jose Castillo Lema <[email protected]>

akrzos

LGTM

mcornea · 2025-10-14T07:40:47Z

/lgtm

mcornea · 2025-10-14T07:41:07Z

/approve

openshift-ci · 2025-10-14T07:41:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mcornea

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mcornea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot requested review from akrzos and jtaleric September 27, 2025 10:48

openshift-ci bot assigned akrzos Oct 2, 2025

josecastillolema added the enhancement New feature or request label Oct 2, 2025

akrzos requested changes Oct 3, 2025

View reviewed changes

openshift-merge-robot added the needs-rebase label Oct 8, 2025

josecastillolema added 17 commits October 13, 2025 14:44

[PERFSCALE-3943] LACP bonding for perf/scale lab (2)

38ca82b

Signed-off-by: Jose Castillo Lema <[email protected]>

Use ocpinventory.json instead of foreman info

550c619

Signed-off-by: Jose Castillo Lema <[email protected]>

Set bastion_bond0_interfaces in create-inventory

896a0b5

Signed-off-by: Jose Castillo Lema <[email protected]>

Fixed nmcli bond setup

df75c1e

Signed-off-by: Jose Castillo Lema <[email protected]>

Added bastion bond interface configuration to both inventory templates

9e8f422

Signed-off-by: Jose Castillo Lema <[email protected]>

wait-hosts-discovered-set the hostname bonded scenario

3a44fec

Signed-off-by: Jose Castillo Lema <[email protected]>

Add temporary debug task

0cf2d70

Signed-off-by: Jose Castillo Lema <[email protected]>

Fix bond port numbering

f254b08

Signed-off-by: Jose Castillo Lema <[email protected]>

Temporary debug task

ad177fa

Signed-off-by: Jose Castillo Lema <[email protected]>

Fixed inventory generation

5e9f087

Signed-off-by: Jose Castillo Lema <[email protected]>

Fix MNO bonding mac assignment logic

3918cb2

Signed-off-by: Jose Castillo Lema <[email protected]>

Address comments

473d50b

Signed-off-by: Jose Castillo Lema <[email protected]>

Address comments (2)

3afbbaf

Signed-off-by: Jose Castillo Lema <[email protected]>

Address comments (3)

2632801

Signed-off-by: Jose Castillo Lema <[email protected]>

Address comments (4)

da1555b

Signed-off-by: Jose Castillo Lema <[email protected]>

Address comments (5)

18bd8cd

Signed-off-by: Jose Castillo Lema <[email protected]>

Fix extra space

d290c0f

Signed-off-by: Jose Castillo Lema <[email protected]>

josecastillolema force-pushed the bond3 branch from 86481f7 to d290c0f Compare October 13, 2025 12:44

openshift-merge-robot removed the needs-rebase label Oct 13, 2025

akrzos reviewed Oct 13, 2025

View reviewed changes

akrzos requested a review from mcornea October 13, 2025 14:09

openshift-ci bot assigned mcornea Oct 14, 2025

openshift-ci bot added the lgtm label Oct 14, 2025

openshift-ci bot added the approved label Oct 14, 2025

josecastillolema merged commit 0aecd9c into redhat-performance:main Oct 14, 2025
1 check passed

Uh oh!

[PERFSCALE-3943] LACP bonding for perf/scale lab (2) #706

[PERFSCALE-3943] LACP bonding for perf/scale lab (2) #706

Uh oh!

Conversation

josecastillolema commented Sep 27, 2025

Uh oh!

josecastillolema commented Sep 29, 2025

Uh oh!

josecastillolema commented Oct 2, 2025

Uh oh!

josecastillolema commented Oct 2, 2025

Uh oh!

josecastillolema commented Oct 2, 2025

Uh oh!

akrzos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

akrzos Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

akrzos Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

akrzos left a comment

Choose a reason for hiding this comment

Uh oh!

mcornea commented Oct 14, 2025

Uh oh!

mcornea commented Oct 14, 2025

Uh oh!

openshift-ci bot commented Oct 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants